Papers with reading comprehension datasets
Generative Data Augmentation using LLMs improves Distributional Robustness in Question Answering (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing domain adaptation methods do not account for unseen natural distribution shifts. |
| Approach: | They perform experiments on 4 different datasets under varying amounts of distribution shift . they analyze how "in-the-wild" generation can help achieve domain generalization . |
| Outcome: | The proposed approach augments reading comprehension datasets with generated data to improve robustness towards natural distribution shifts. |
IIRC: A Dataset of Incomplete Information Reading Comprehension Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing reading comprehension tasks focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system’s performance at identifying a potential lack of sufficient information and locating sources for that information. |
| Approach: | They propose to use a dataset with 13K questions over paragraphs from English Wikipedia that provide only partial information to answer them, with the missing information occurring in one or more linked documents. |
| Outcome: | The proposed model achieves 31.1% F1 on the reading comprehension task, while estimated human performance is 88.4%. |
MultiSpanQA: A Dataset for Multi-Span Question Answering (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing reading comprehension datasets focus on single-span answers, but multi-spread questions are less studied. |
| Approach: | They propose a new reading comprehension dataset that focuses on multi-span questions . they introduce new metrics for the purposes of multi--spontaneous question answering evaluation . |
| Outcome: | The proposed model beats baselines and achieves state-of-the-art on the existing dataset. |
Generating Question-Answer Hierarchies (P19-1)
Copied to clipboard
| Challenge: | a novel text generation task uses reading comprehension datasets to generate a hierarchy of question-answer pairs . users can click on high-level questions to reveal related but more specific questions . |
| Approach: | They propose a text-generating task which converts a document into a hierarchy of question-answer pairs . users can click on high-level questions to reveal related but more specific questions . they then use the specificity-labeled reading comprehension dataset to generate the hierarchy . |
| Outcome: | The proposed system can be used to generate a hierarchy of question-answer pairs based on reading comprehension datasets. |
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning (D19-1)
Copied to clipboard
| Challenge: | Existing reading comprehension datasets focus on factual and literal understanding of context paragraphs, but our dataset focuses on reading between the lines over a diverse collection of everyday narratives. |
| Approach: | They propose a large-scale dataset that requires commonsense-based reading comprehension, formulated as multiple-choice questions. |
| Outcome: | The proposed architecture improves over the baselines of existing reading comprehension datasets and shows a significant gap between machine (68.4%) and human performance (94%). |
BiPaR: A Bilingual Parallel Dataset for Multilingual and Cross-lingual Reading Comprehension on Novels (D19-1)
Copied to clipboard
| Challenge: | Using BiPaR, we build monolingual, multilingual and cross-lingual MRC on novels. |
| Approach: | They propose a bilingual parallel novel-style machine reading comprehension dataset BiPaR . they collect 3,667 bilingual parallel paragraphs from Chinese and English novels . |
| Outcome: | The proposed dataset supports multilingual and cross-lingual reading comprehension. |
Dataset for the First Evaluation on Chinese Machine Reading Comprehension (L18-1)
Copied to clipboard
| Challenge: | Existing reading comprehension datasets are mostly in English . |
| Approach: | They propose a Chinese reading comprehension dataset to add diversity to existing reading comprehension data . proposed dataset contains cloze-style reading comprehension and user query reading comprehension . |
| Outcome: | The proposed dataset is based on a Chinese reading comprehension dataset . it includes two types of cloze-style and user query reading comprehension . the proposed dataset hosted the 1st Evaluation on Chinese Machine Reading Comprehension (CMRC-2017) |
A Span-Extraction Dataset for Chinese Machine Reading Comprehension (D19-1)
Copied to clipboard
| Challenge: | Existing reading comprehension datasets are mostly in English . MRC is a new field of research that aims to comprehend the context of articles and answer the questions based on them. |
| Approach: | They propose a Span-Extraction dataset for Chinese machine reading comprehension to add language diversities to existing reading comprehension datasets. |
| Outcome: | The proposed dataset is composed of 20,000 real questions annotated on Wikipedia paragraphs by human experts. |
Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on citation generation has focused on unambiguous settings with single answers, failing to address the complexity of real-world scenarios. |
| Approach: | They propose a task of QA with source citation in ambiguous settings where multiple valid answers exist, where multiple sources exist. |
| Outcome: | The proposed framework generates multiple answers and cites their sources, allowing users to verify the factuality of each answer and make informed decisions. |